1 

James-Stein Type Center Pixel Weights for 
Non-Local Means Image Denoising 

Yue Wu, Member, Brian Tracey, Senior Member, and Joseph P. Noonan, Life Member, IEEE, 

Abstract 

Non-Local Means (NLM) and variants have been proven to be effective and robust in many image 
denoising tasks. In this letter, we study the parameter selection problem of center pixel weights (CPW) in 
NLM. Our key contributions are: 1) we give a novel formulation of the CPW problem from the statistical 
shrinkage perspective; 2) we introduce the James-Stein type CPWs for NLM; and 3) we propose a new 
adaptive CPW that is locally tuned for each image pixel. Our experimental results showed that compared 
to existing CPW solutions, the new proposed CPWs are more robust and effective under various noise 
levels. In particular, the NLM with the James-Stein type CPWs attain higher means with smaller variances 
in terms of the peak signal and noise ratio, implying they improve the NLM robustness and make it less 
sensitive to parameter selection. 

Index Terms 

Image Denoising, Non-Local Means, Adaptive Algorithm, Shrinkage Estimator, James-Stein Estima- 
tor 

I. Introduction 

Image noise commonly exists in the image acquisition, quantization, transmission and many other 
processing stages. A digital image contaminated by noises leads to visible loss in image quality and 
can impact many advanced image processing and computer vision tasks like tracking, recognition, 
classification, etc. The importance of image denoising is therefore self-explanatory. 

Conventional image denoising methods more or less related to filters |1], like moving average filters, 
Wiener filters, and wavelet filter banks. These filter-based image denoising techniques are commonly of 
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low complexity and can be easily implemented in hardware. However, their performance is not always 
adequate. With the help of the increased computational capacity powered by digital processors, many 
advanced denoising techniques are now feasible. Among these techniques, the Non-Local Means (NLM) 
method |TJ, (2j has attracted significant attention in recent years. The NLM denoises an image pixel 
as the weighted sum of its noisy neighbors, where each weight reflects the similarity between the local 
patch centered at the noisy pixel to be denoised and the patch centered at the neighbor pixel. In this way, 
NLM adapts the denoising process for each pixel and thus outperforms conventional techniques |T|. 

Many improvements on the original NLM have been proposed in recent years. These discussions mainly 
focus on three questions: 1) how to pick NLM parameters heuristically or automatically (3j, Q; 2) how 
to accelerate the NLM or save computations without loss of denoising performance |5)-|7); and 3) how 
to adjust the NLM framework to achieve better performance |8|-|T2j. In j3), the closed-form of Stein's 
risk estimator is derived for the NLM and allows prediction of the risk of a denoised image without 
knowing the clean image. In Q, the integral image scheme is adapted to the NLM and greatly reduces 
the computational costs. In jl2j , multiple-patches are discussed to eliminate artifacts in the NLM. 

The importance of the center pixel weight (CPW) has been noticed for a long time [T|, |4}, and various 
heuristic weights are designed and used Q. However, the methods proposed are non-ideal as they do 
not consider all aspects of CPW problem (see details in Section III-A). Thus, new CPWs need to be 
designed. In this letter, we discuss the CPW problem in NLM and propose new solutions based on the 
James-Stein estimator [13]. The rest of the letter is organized as: Sec. II reviews the NLM and related 
works on CPWs; Sec. Ill discusses the new formulation of the CPW problem and the new James-Stein 
type CPWs; Sec. IV shows the experimental comparisons; and we conclude the letter in Sec. V. 

II. A Brief Review 

A. The Classic Non-Local Mean Algorithm 

Let x = {x;}z e i be a two dimensional clean image defined on the spatial domain I, with / = 
the Zth pixel located at the intersection of the l\th row and the l$Qi column. Given one of x's noisy 
observation y = {*/;};ei, whose pixels are contaminated by an i.i.d. zero-mean Gaussian noise with a 
variance of a 2 , namely 

yi=xi + ni, and ni ~ N(0,a 2 ). (1) 

The classic NLM method |TJ, (2J estimates the clean pixel x\ by using all pixels within a prescribed 
search region §, typically a square or a rectangular region. Specifically, the estimated xi is the weighted 
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sum of yi's noisy neighbors as 

^ (2) 

where each weight is computed by quantifying the similarity between two local patches around noisy 
pixels m and y k as shown in Eq. ([3]), where G a is a Gaussian weakly smooth kernel |1], P denotes the 
local patch, typically a square centered at the pixel (for example a P = [—1, 0, 1] x [—1, 0, 1] denotes a 
3x3 patch). The parameter h can be considered as a temperature parameter controlling the behavior of 
the weight function, namely as h— >0, all weights approach to 1; while as /i— >oo, all weights approach 
to zero. 

wi,k = exp (-^2 je¥ G a (yi +j - y k+j ) 2 /h)} (3) 

B. Existing Central Pixel Weights 

The CPW in the classic NLM is unitary, because ([3]) implies wu = l for all I el. However, this unitary 
CPW is reported not to perform well in many cases |4|. Indeed, in the case that yi is really noisy and yi 
has a rare patch (implying that for all k e S\/ non-center weights w^k are very small compared to 1), a 
unitary CPW means the contribution of the noisy pixel y\ dominates in the denoised pixel xi, implying 
a poor performance. 

In addition to this CPW, several other CPWs have been proposed and used in the NLM community 
to enhance performance. These include the zero CPW (Eq. ([5])), the Stein CPW (Eq. (|6])), the max CPW 
(Eq. (|7])) and the heuristic CPW (Eq. ([8])). In the rest of the letter, we use vi to denote these existing 
CPWs. 

vf ne = 1 (4) 
vf ero = (5) 
vf tein = ex V (-a 2 \P\/h) (6) 
vr x = ™*w lM (7) 

, oo, if vV iax < threshold 

I y max ^ otherwise 

These CPWs can be classified into two groups: global CPWs (Eqs. (|4]),((5]) and Q) and local CPWs 
(Eqs. {7]) and The global CPWs use a constant CPW for all pixels, while the local CPWs use 
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different CPWs for pixels. In the next section, we will show that these CPWs fail to take all variables 
into full consideration and therefore oversimplify the CPW problem. 

III. Shrinkage based Center Pixel Weights 

A. The CPW problem in the form of shrinkage estimator 

To fully reveal the CPW problem, we separate the contributions of the non-center and of the center 
pixels in the NLM denoised pixel xi (Eq. Q) 

Wi + vi Wi + vi 
where W\ is the summation of all non-center weights 

w i = Efces\l w Lk (10) 
and % is the denoised pixel by using all non-center weights. 

= J2kes\i w i,kVk/Wi. (11) 

As one can see, a CPW v\ is contained in the coefficients of both % and yi and thus influences the 
final denoised pixel x\ directly. If we are given an optimal x\ and solve for vi, then it is clear that the 
optimal vi should be a function of Wi, % and y\. In other words, a CPW v\ that does not consider all 
these variables is incomplete. It is noticeable that the global CPWs v° ne vf ero and vf tem neglects all 
three, while the local CPWs vf iax and vf eur ' neglect yi. 

Let pi be the fraction (pi £ [0, 1]) of the contribution of the center pixel yi in xi, namely 

Pi = vi/{vi + Wi). (12) 
Pi is then a normalized version of vi. Consequently, the NLM-CPW problem in ([9]) can be rewritten as 

xi = (1 - pi)zi + piyi (13) 

and is a so-called shrinkage estimator, which improves an existing estimator by using the raw data. In 
the context of the NLM, the existing estimator is % and the raw data is the noisy pixel yi. The effect of 
the CPW is to tune the final denoised pixel x~i in somewhere between % and yi, or equivalently to shrink 
yi towards to z\. 



November 8, 2012 



DRAFT 



B. The James-Stein Center Pixel Weight 

One important result in shrinkage estimators is the James-Stein estimator (13). It states that for an 
unknown parameter vector a and its observations of b with the relation, 

b~AA(a,a 2 I) (14) 

there exists a James-Stein estimator that shrinks towards an arbitrary vector c in the form that 

a JS = c + q(b-c) = (l-g)c + gb (15) 

with the coefficient q of form ( fT7[ ) p3| . 

q= l-(m-2)cr 2 /||b-c|| 2 (16) 

This James-Stein estimator is a classic solution to minimize the risk of estimation in terms of the mean 



square error i?[||a— a|| ] [14|, where ||.|| denotes the L -norm. 



In the context of NLM-CPW problem, the James-Stein based CPW (JSCPW) has the weight of form 



p* JS = l-(rn-2)a 2 /\\y-zf (17) 
where m=|I| is the number of pixels in the image, and the corresponding new estimator is 

x JS = (1 - p * JS )z + p* Js y. (18) 

C. Local Adapted James-Stein Center Pixel Weights 

Although the proposed JSCPW considers all Wi, xi and yi, it is still a global CPW, which gives the 
weight to each pixel unbiasedly. However, we know the denoising process is always biased rather than 
unbiased for each pixel. For example, pixels in a homogeneous region are commonly better denoised than 



edge pixels. Therefore, ideally we want a locally adapted CPW for each pixel as pi in the form of ([20 1. 
One natural idea is to replace ||y — z|| 2 in ( |20| ) with \\yi — zi\\ 2 , but it does not lead to a stable solution, 
because \\yi — zi\\ 2 is commonly noisy. Alternatively, we view each image block as a small image and 
thus the JSCPW ( [17] ) computed for a local block gives a local CPW adapted to each pixel. 

Without loss of generality, say zb; = {z^|V6 G B} and yb ; = {y/+&|V6 e B}, so there are two 
corresponding local image blocks around the Zth pixel in z and y, respectively. Given a prescribed local 
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block region B, then the local James-Stein center pixel weight (LJSCPW) can be computed as 

Pl US = 1 " (|B| - 2)a 2 /||yb i - zbi\\ 2 . (19) 

In this way, we construct a local CPW for each pixel, and thus the denoised pixel by using LJSCPW 
can be written as 

x t LJS = (l-p! LJS )?i+p? LJS yi. (20) 

Intuitively, this LJSCPW helps eliminate the influence of remote image pixels and tunes the optimization 
locally. 

D. Implementation 

The computational cost of the LJSCPW can be only 5 operations/pixel more than the existing CPWs. 
Specifically, we construct the integral image |5] II with 2 operations/pixel for the pixel- wise mean square 
error between z and y. Each pixel is the summation of form ( |2Tj ). 

n i = Si={(i 1) i a )|i I 6[l,ii],i 2 e[i,i2]}(^ ~~ Zi ) 2 ' ( 21 ) 

This integral image II then allows computation of |tyb ; — zb;|| 2 for an arbitrary rectangular B with 3 
operations/pixel. 

In a real case, one may see p* LJS < which conflicts with our assumption that p* LJS £ [0, 1]. This 
case happens when zr; is a slightly denoised version of yr ; . So it is reasonable to use Z\ rather than yi, 
implying p* LJS = 0. Therefore, we use the positive part of p* LJS in Eq. (22 1, where (.) + = max(., 0). 



p *LJS+ m _ 2)(J 2 /||yb / - zb,|| 2 ) (22) 

Thus p* l LJS+ ranges from [0, 1]. Because p* LJS+ = i would indicate the raw data is used, i.e. the pixel 
is not denoised, it may prove useful in some applications to limit p* l LJS+ to a user-defined value less 
than unity. In this letter, however, we allow the shrinkage operator to operate over the full range. 

IV. Simulation Results 

All following simulations are done under the MATLAB r2010a environment. We compare the perfor- 
mance of CPWs i.e. v° ne ,vf ero , v f ein , v ™ ax ,v\ eur - with the proposed James-Stein type CPWs p* JS and 
p*LJS unc i er the classic NLM framework (only CPW part is changed). In particular, we set searching 
region S to 31x31 square and use B = P for p* LJS in the experiment. All test images 1 are gray-scale 



November 8, 2012 



DRAFT 



7 



images with additive Gaussian noises of ct€ {10, 20, 40}. For each test image, we denoise it by using the 
200 temperature parameters h ranging from 1% to 200% of a 2 |P|. The denoising performance for each 
CPW scheme is then evaluated by computing its mean and standard deviation in terms of the peak signal 
noise ratio (PSNR) (dB) defined in Eq. p3| ). 

PSNR(x, x) = 20 log 10 255 - 10 log 10 ||x - x|| 2 /|I| (23) 

These results are summarized in Table [I] As one can see, compared to the PSNR performance of the 
zero CPW, the proposed pl LJS (LJSCPW) and p* JS (JSCPW) are the only two in all CPWs that always 
improve the denoising performance in terms of a higher mean PSNR with a smaller variance, regardless 
of patch sizes, test images and noise levels. This implies that James-Stein type CPWs are more efficient 
than other CPWs. 



TABLE I: (mean± standard deviation) PSNR (dB) comparisons for various center pixel weighting schemes 
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a 


256 x 256 cameraman 


10 


28.14±2.73 


31.72±1.07 


31.69±0.80 


30.92±0.75 


31.22±1.03 


30.70±0.97 


32.75±0.56 


26.18±3.41 


31.27±1.08 


31.38±0.80 


30.03±0.65 


30.61±1.12 


30.26±0.90 


32.59±0.50 


20 


27.25±1.28 


27.73±1.61 


27.93±1.05 


27.74±1.02 


27.75 ±1.05 


27.66±1.08 


28.50±0.71 


2S.31± 1.62 


27.30±1.67 


27.69± 1.13 


27.28±1.09 


27.28±1.23 


27.26±1.05 


28.57±0.66 


40 


23.76±1.17 


23.35±2.03 


23.79±1.18 


23.75±1.27 


23.74±1.29 


23.90±1.08 


24.10±0.96 


23.33±1.28 


22.77±2.20 


23.44± 1.31 


23.43±1.32 


23.38±1.46 


23.69±1.07 


24.31 ±0.87 


a 


256x256 house 


10 


33.30±2.03 


33.68±1.58 


34.08±0.94 


33.97±0.94 


33.97±0.99 


33.77±1.36 


34.34±0.73 


32.73±2.68 


33.34±1.70 


34.01 ±0.88 


33.86±0.89 


33.82±1.09 


33.69±1.36 


34.45±0.69 


20 


30.07±1.20 


29.60±2.10 


30.11±1.18 


30.06±1.27 


30.05±1.30 


30.16±1.14 


30.09±0.98 


29.87±1.49 


29.18±2.34 


30.03±1.38 


30.01±1.38 


29.95±1.55 


30.15± 1.33 


30.47±1.00 


40 


25.50±1.41 


25.10±2.47 


25.67±l.40 


25.57±1.56 


25.57±1.57 


25.70±1.38 


25.68±1.17 


25.78± 1.55 


24.79±2.78 


25.79± 1.55 


25.72±1.59 


25.68il.75 


25.92±1.44 


25.97±1.24 


a 


256 x 256 peppers 


10 


30.41±2.21 


31.98±1.26 


32.13±0.91 


31.80±0.86 


31.87 ±0.94 


31.65±1.0O 


32.75±0.60 


29.15±3.11 


31.60±1.30 


32.03±0.93 


31.37 ±0.88 


31 48± 1.12 


31.55±1.01 


32.61±0.62 


20 


27.74±1.12 


27.54±1.68 


27.92±1.09 


27.89±1.10 


27.88±1.13 


28.04±0.95 


28.38±0.80 


27.16±1.49 


27.06±1.83 


27.64±1.31 


27.59±1.27 


27.53±1.41 


27.86±1.02 


28.41 ±0.84 


40 


23.53±1.25 


23.04±2.01 


23.55±1.25 


23.48±1.32 


23.48±1.34 


23.68±1.13 


23.67±0.98 


23.38±1.46 


22.60±2.26 


23.41 ±1.46 


23.39±1.46 


23.34±1.61 


23.73±1.20 


23.89±1.08 


a 


512x512 lenna 


10 


32.75±1.58 


32.90±1.45 


33.18±0.94 


33.19±0.94 


33.18±0.97 


33.22±1.06 


33.65±0.67 


32.02±2.36 


32.57±1.57 


33.09±1.02 


33.01 ±1.03 


32.95±1.18 


33.13±1.13 


33.83±0.67 


20 


29.64±1.05 


29.16±1.96 


29.66±1.04 


29.61±1.14 


29.60±1.16 


29.73±0.99 


29.67±0.89 


29.40±1.20 


28.73±2.17 


29.50± 1.14 


29.50±1.15 


29.44±1.34 


29.71 ±1.05 


29.97±0.86 


40 


25.96±1.31 


25.40±2.48 


25.96±1.31 


25.86±1.50 


25.8fiil.51 


25.96±1.31 


25.96±1.13 


26.21±1.25 


25.25±2.86 


26.22±1.25 


26.15±1.36 


26.1 1± 1.55 


26.30±1.19 


26.23±1.11 


a 


512x512 barbara 


10 


31.44±1.83 


31.84±1.28 


32.04±0.96 


32.10±0.93 


32.09±0.97 


32. 15 ±0.92 


32.69±0.62 


30.62±2.83 


31.62±1.37 


32.07±1.02 


31.97 ±0.98 


31.91±1.11 


32.24±1.01 


32.90±0.65 


20 


27.74±1.17 


27.29±1.66 


27.77±1.16 


27.75±1.17 


27.74±1.20 


28.02±0.95 


28.13±0.85 


27.70±1.44 


27.08±1.88 


27.82±1.39 


27.86±1.39 


27.79±1.52 


28.27±1.10 


28.56±0.93 


40 


23.72±1.09 


23.21±1.96 


23.72±1.09 


23.64±1.19 


23.73±1.20 


23.82±1.01 


23.77±0.93 


23.92±1.28 


23.05±2.30 


23.93±1.28 


23.88±1.29 


23.83±1.47 


24.20±1.10 


24. 15±1.03 


a 


512x512 boat 


10 


30.30±1.98 


31.15±1.12 


31.25±0.96 


3 1.21 ±0.97 


31.21±1.00 


31.50±0.87 


32.14±0.5I 


29.22±2.75 


30.68±1.08 


30.90±1.02 


30.73±1.07 


30.70±1.15 


31.47±0.86 


32.08±0.52 


20 


27.54±1.06 


27.26±1.57 


27.61±i.05 


27.61 ±1.07 


27.60±1.10 


27.90±0.84 


28.12±0.73 


26.89±1.34 


26.64±1.64 


27.14±1.20 


27.16±1.22 


27.10±1.33 


27.66±0.91 


28.08 ±0.71 


40 


24.18±1.05 


23.70±2.03 


24.19±1.05 


24.12±1.18 


24.1 1± 1.19 


24.26±1.00 


24.25±0.91 


24.03 ±1.09 


23.27±2.26 


24.04±1.09 


24.01±1.13 


23.97±1.31 


24.30±0.94 


24.38±0.86 



We found two general performance trends: 1) as /i— )-oo, all CPWs except p* LJS and p* JS converges 
together; and 2) as a increases, the impact of CPWs on denoising results decreases. Fig. 1 illustrates these 
trends in NLM performance for the cameraman image. The first trend occurs because as h — > oo, all 
non-center weights go to 1. The CPWs v° ne , vf em , v™ ax and v^ eur ' are then all Is and thus converge. 
The second trend is because the noisy image y become less effective in providing a good guide in 
shrinkage when the noise level is high. Thus the performance differences of various CPWs gets smaller 
as the noise level increases. However, it is noticeable that the proposed locally adapted pJ LJS outperforms 



are available under the page http://www.cs.tut.fi/~foi/GCF-BM3D/BM3D_images.zip 
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other CPWs regardless of chosen temperature parameter h, and the NLM performance using the proposed 
CPWs decays much slower than using other CPWs. Similar behaviors are also observed for other patches 
sizes and test images. Finally, the NLM denoising results for h = a 2 \F\ is given in Fig. 2. It is noticeable 
that the proposed p* LJS helps keep weak edges (see the two buildings in the background). 

V. Conclusion 

In this letter, we reviewed the CPW problem in NLM and proposed two new solutions JSCPW and 
LJSCPW based on the James-Stein estimator. We showed that the NLM-CPW problem can be viewed 
as the well studied statistical shrinkage estimator problem. This novel formulation opens a new door to 
the CPW problem and allows us to use the James-Stein shrinkage estimator for the NLM-CPW problem 
directly, that is the global JSCPW. To further enhance denoising performance, we propose a locally 
adapted James-Stein type CPW for each pixel. In this way, the denoising performance is tuned with 
respect to each local pixel rather than an entire image. Our experimental results show that the proposed 
James-Stein type CPWs help the NLM to achieve better overall performance in terms of a higher average 
PSNR with a more robust performance in terms of a smaller PSNR variance. By using these new CPWs, 
the NLM algorithm is then less sensitive to the temperature parameter h and is better able to retain weak 
edges. 
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Fig. 1: The PSNR (dB) performance for various CPWs 
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Fig. 2: NLM denoising using various CPWs (h = a 2 \V\, 7x7 P). 
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